test(consumer): per-method broker for RebalanceAtScaleIntegrationTest — fix CI flake#224
Merged
Merged
Conversation
…ker to fix CI flake RebalanceAtScaleIntegrationTest was the only class failing on main CI (3 runs: the #221 merge run + both attempts of the #218 run). The signature was identical every time: produce timed out at line 325 (send.get in the produceRecords setup helper) with org.apache.kafka.common.errors.TimeoutException, always in this one class, for its later-ordered methods (cooperativeSticky, revoke) while midFlight passed. Root cause: the class shared ONE static KafkaContainer across its three @test methods, each of which creates its own 6-partition topic + consumer groups and never tears them down. That state accumulated on the single broker until a later method's producer could not get acks within delivery.timeout.ms, failing at the produce step during setup. It is the only container test class heavy enough (3 methods x 6 partitions) to degrade its own broker this way, which is why it was the only one that flaked — a runner-wide resource problem would have scattered failures across the other five container classes, and it did not. Fix: make the @container field non-static so JUnit's default per-method lifecycle starts a fresh broker for each test, isolating them. Costs ~2 extra broker startups (~40s) for this one class; leaves the five healthy container classes untouched. Cannot run Testcontainers locally (no Docker); verified it compiles. CI is the confirmation — will re-run the branch a few times to confirm the rebalance tests stay green.
Owner
Author
|
@copilot please review |
Reviewed the current patch and CI context for this change. No additional code changes are needed for this comment. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #224 +/- ##
=======================================
Coverage ? 80.52%
Complexity ? 794
=======================================
Files ? 66
Lines ? 2927
Branches ? 372
=======================================
Hits ? 2357
Misses ? 411
Partials ? 159 ☔ View full report in Codecov by Harness. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
RebalanceAtScaleIntegrationTestis the only test failing onmainCI. Across 3 failing runs (the #221-merge run + both attempts of the #218upload-artifactrun) the signature is identical:Line 325 is
send.get(30s)in theproduceRecords()setup helper — the producer can't get acks. Always this one class, always at produce, always the later-ordered methods (midFlightpasses).Root cause
The class shared one static
KafkaContaineracross its three@Testmethods. Each method creates its own 6-partition topic + consumer groups and never tears them down, so that state accumulates on the single broker until a later method's producer can't deliver withindelivery.timeout.msand fails at the produce step.It's the only container test class heavy enough (3 methods × 6 partitions) to degrade its own broker this way — which is exactly why it's the only class that flakes. A runner-wide resource problem would scatter failures across the other five container classes; it doesn't (ruling out the "too many forks / not enough RAM" theory).
Not caused by the
upload-artifactv4→v7 bump, and not by the kafka 4.3.1 client bump.Fix
Make the
@Containerfield non-static so JUnit's default per-method lifecycle starts a fresh broker per test, isolating them. A why-comment on the field documents this so it isn't "optimized" back to static.RebalanceAtScaleIntegrationTest; the five healthy container classes are untouched.Verification
Testcontainers can't run locally (no Docker) — verified it compiles. CI is the confirmation; I'll re-run this branch a few times to confirm the rebalance tests stay green before merging.